Image processing apparatus, image processing method, and storage medium

ABSTRACT

An image processing apparatus includes: an imaging device including an optical system for forming an optical image including a low-distortion region and a high-distortion region on a light receiving surface and configured to generate an image signal of a side behind a moving apparatus; an image processing unit configured to generate image data from the image signal generated by the imaging device; a detection unit configured to detect another moving apparatus on a lateral side behind the moving apparatus; and a display control unit configured to cause the image data in a predetermined cut region including the high-distortion region including the another moving apparatus behind on the lateral side to be displayed in a case where the another moving apparatus behind on the lateral side is detected by the detection unit.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an image processing apparatus, an imageprocessing method, and a storage medium capable of displaying lateralsides behind a moving apparatus.

Description of the Related Art

WO18/207393 describes a method of capturing a wide-angle image at thetime of backward traveling and capturing a high-resolution,high-frame-rate image of a side behind a vehicle during ordinarytraveling with a narrower angle of view than the wide angle, using animaging device adapted to acquire a plurality of angles of view.

However, WO18/207393 has a problem that it is not possible to takeadvantage of most of the acquired image at the wide angle since only theside behind the vehicle itself is the target of imaging during ordinarytraveling while an image at the wide angle is acquired at the time ofbackward traveling. Additionally, there is a problem that it is not easyto visually recognize the lateral sides behind the vehicle itself withside mirrors.

SUMMARY OF THE INVENTION

In order to solve the aforementioned problem, an image processingapparatus according to an aspect of the present invention includes: animaging device including an optical system for forming an optical imageincluding a low-distortion region and a high-distortion region on alight receiving surface and configured to generate an image signal of aside behind a moving apparatus; and at least one processor or circuitconfigured to function as: an image processing unit configured togenerate image data from the image signal generated by the imagingdevice; a detection unit configured to detect another moving apparatuson a lateral side behind the moving apparatus; and a display controlunit configured to cause the image data in a predetermined cut regionincluding the high-distortion region including the another movingapparatus behind on the lateral side to be displayed in a case where theanother moving apparatus behind on the lateral side is detected by thedetection unit.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram for explaining a configuration example of animage processing apparatus 200 according to a first embodiment.

FIG. 2A is a diagram illustrating, as a contour line, an image height yat each half angle of view on a light receiving surface of an imagingelement of an optical system 10, and FIG. 2B is a diagram representing aprojection property representing a relationship between the image heighty and a half angle of view θ of the optical system 10.

FIG. 3A is an image diagram illustrating a first region R1 and a secondregion R2, and FIG. 3B is an image diagram illustrating a third regionR3 and the second region R2.

FIG. 4 is a flowchart illustrating control of changing a position wherean image is to be cutout when a vehicle 310 is detected on a lateralside behind the vehicle itself from an acquired image according to thefirst embodiment.

FIG. 5 is a block diagram for explaining a configuration example of animage processing apparatus according to a second embodiment.

FIG. 6 is a block diagram for explaining a configuration example of animage processing apparatus according to a third embodiment.

FIG. 7 is a block diagram for explaining a configuration example of animage processing apparatus according to a fourth embodiment.

FIG. 8 is a flowchart illustrating control for changing a cuttingposition in an image when an operation of a direction indicator has beendetected according to the fourth embodiment.

FIG. 9 is a block diagram for explaining a configuration example of animage processing apparatus according to a fifth embodiment.

FIG. 10 is a flowchart for controlling whether or not to change aposition to be cut in an image in accordance with setting of an imagecutting setting saving unit 120 according to the fifth embodiment.

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, with reference to the accompanying drawings, favorablemodes of the present invention will be described using Embodiments. Ineach diagram, the same reference signs are applied to the same membersor elements, and duplicate description will be omitted or simplified.

First Embodiment

Hereinafter, a first embodiment will be described in detail withreference to the drawings. It is necessary to use an imaging device forcapturing an image behind on a lateral side and a detection unit fordetecting a vehicle 310 behind on the lateral side in order to check avehicle at a blind angle part where the vehicle is difficult to bechecked with a side mirror, such as a lateral side behind the vehicleitself. In the first embodiment, a method for displaying a vehicle,which is another moving apparatus, in a blind angle part, such as on alateral side behind the vehicle, in which it is difficult to see avehicle with a side mirror, merely by using an imaging apparatus forimaging the side behind the vehicle itself will be described.

FIG. 1 is a block diagram for explaining a configuration example of animage processing apparatus 200 according to the first embodiment. Notethat one or more components illustrated in FIG. 1 may be realized byhardware such as an ASIC or a programmable logic array (PLA).

The same applies to FIGS. 5, 6 . 7, and 9, which will be describedlater. A computer (such as a central processing unit (CPU) or amicrocomputer) included in the image processing apparatus 200 functionsas a control unit for controlling operations of each component in theapparatus 200 on the basis of a computer program stored in a storagemedium.

In the first embodiment, the image processing apparatus 200 is an imageprocessing unit mounted on a rear side of the vehicle (moving apparatus)itself, which is not illustrated, and configured to display an imagecaptured by an imaging device for capturing a rear-side image on adisplay device in the vehicle, for example.

The image processing apparatus 200 includes an optical system 10, animaging unit 20, an image processing unit 30, an image cuttingprocessing unit 40, an image cutting position changing unit 50, a rearmonitor 60, an electronic rear-view mirror 70, and avehicle-behind-on-lateral-side recognition unit 80. Note that theoptical system 10 and the imaging unit 20 are mounted to face the sidebehind the vehicle itself to image the side behind the vehicle itself asa moving apparatus. The electronic rear-view mirror 70 is a displaydevice such as a LCD or an OLED for displaying a rear-view image.

The optical system 10 is configured to be able to obtain ahigh-definition image at a narrow angle of view 11 a around an opticalaxis using a combination of a plurality of lenses, for example, and beable to obtain an image captured with low resolution with a wide angleof view 11 b including the narrow angle of view 11 a. For example, theoptical system 10 can form an optical image including a low-distortionregion and a high-distortion region on a light receiving surface of theimaging unit 20.

The wide angle of view 11 b is suitable for imaging the side behind thevehicle itself at a wide angle at the time of backward traveling or thelike. Also, it is possible to image a vehicle on the lateral side behindthe vehicle itself at a part corresponding to a blind angle where it isdifficult to recognize the vehicle with a side mirror. The narrow angleof view 11 a can be used to observe a vehicle 300 on the side behind thevehicle itself with high resolution at the time of ordinary traveling.

The imaging unit 20 is an imaging element (such as a CCD image sensor ora CMOS image sensor), converts an optical object image formed by theoptical system 10 into imaged signal, and transmits the image data tothe image processing unit 30.

For example, the imaging unit 20 including the optical system functionsas an imaging device that executes an imaging step of imaging the sidebehind the moving apparatus using the optical system that forms anoptical image including a low-distortion region and a high-distortionregion on the light receiving surface.

Optical properties of the optical system 10 will be described withreference to FIGS. 2A and 2B. FIG. 2A is a diagram illustrating, as acontour line, an image height y at each half angle of view on the lightreceiving surface of the imaging element included in the imaging unit20.

FIG. 2B is a diagram representing a projection property representing arelationship between the image height y and the half angle of view θ ofthe optical system 10. In FIG. 2B, the half angle of view (the angleformed by the optical axis and the incident light beam) θ is representedby the horizontal axis, and the image height y on the light receivingsurface (image surface) of the imaging element included in the imagingunit 20 is represented by the vertical axis.

The optical system 10 is configured such that the projection propertyy(θ) differs in a region of less than a predetermined half angle of viewθa and a region of equal to or greater than the half angle of view θa asillustrated in FIG. 2B. Therefore, a resolution differs depending on aregion when the amount of increase in image height y with respect to thehalf angle of view θ per unit is referred to as the resolution.

It is also possible to state that the local resolution is represented bya differential value dy(θ)/dθ at the half angle of view θ of theprojection property y(θ). For example, it is possible to state that theresolution increases as a gradient of the projection property y(θ) inFIG. 2B increases. Additionally, it is also possible to state that theresolution increases as the interval of the image height y at each halfangle of view of the contour line in FIG. 2A increases.

In the first embodiment, the region near the center generated on thelight receiving surface of the imaging element when the half angle ofview θ is less than the predetermined half angle of view θa will bereferred to as a high-resolution region 10 a, and the region near theoutside where the half angle of view θ is equal to or greater than thepredetermined half angle of view θa will be referred to as alow-resolution region 10 b.

The angle of view of the high-resolution region 10 a corresponds to theaforementioned narrow angle of view 11 a, and the angle of view obtainedby adding the angle of view of the high-resolution region 10 a and theangle of view of the low-resolution region 10 b corresponds to the wideangle of view 11 b for imaging.

Also, in the first embodiment, the high-resolution region 10 a is alow-distortion region where the amount of distortion is relativelysmall, and the low-resolution region 10 b is a high-distortion regionwhere the amount of distortion is relatively large. Therefore, thehigh-resolution region and the low-resolution region may be referred toas a low-distortion region and a high-distortion region, respectively,in the first embodiment.

The optical system 10 is configured such that the projection propertyy(θ) thereof is greater than f×θ (f is a focal distance of the opticalsystem 10) in the high-resolution region (low-distortion region) 10 a.Also, the projection property y(θ) in the high-resolution region(low-distortion region) is set to be different from the projectionproperty in the low-resolution region (high-distortion region).

In a case where θ max denotes the maximum half angle of view that theoptical system 10 has, it is desirable that a ratio θa/θ max between θaand θ max be equal to or greater than a predetermined lower limit valueand that the predetermined lower limit value be 0.15 to 0.16, forexample.

Also, it is desirable that the ratio θa/θ max between θa and θ max beequal to or less than a predetermined upper limit value and that theupper limit value be 0.25 to 0.35, for example. In a case where θa is90°, the predetermined lower limit value is 0.15, and the predeterminedupper limit value is 0.35, for example, it is desirable to determine θawithin a range of 13.5 to 31.5°.

Furthermore, the optical system 10 is configured such that theprojection property y(θ) also satisfies Expression 1 below.

$\begin{matrix}{1 < \frac{f \times \sin\theta_{\max}}{y\left( \theta_{\max} \right)} \leq A} & \left\lbrack {{Expression}1} \right\rbrack\end{matrix}$

Here, f is a focal distance of the optical system 10 as described above,and A is a predetermined constant. It is possible to obtain highercenter resolution than that of a fisheye lens of the orthographicprojection scheme (y=f×sin θ) having the same maximum image formationheight by setting the lower limit value to 1, and it is possible tomaintain satisfactory optical performance while obtaining an angle ofview that is equivalent to that of the fisheye lens by setting the upperlimit value to A. It is only necessary to determine the predeterminedconstant A in consideration of a balance of the resolution between thehigh-resolution region and the low-resolution region, and it isdesirable that the predetermined constant A be 1.4 to 1.9.

It is possible to obtain high resolution in the high-resolution region10 a and to reduce the amount of increase in image height y with respectto the half angle of view θ per unit and to image a wider angle of viewin the low-resolution region 10 b, by configuring the optical system 10as described above. Therefore, it is possible to achieve the wide angleof view that is equivalent to that of the fisheye lens as an imagingrange and to obtain high resolution in the high-resolution region 10 a.

In the first embodiment, a projection property that is approximated tothose of the center projection scheme (y=f×tan θ) and the equidistanceprojection scheme (y=f×θ) which are projection properties of the opticalsystems for ordinary image capturing in the high-resolution region(low-distortion region) is used.

Therefore, it is possible to achieve high-definition display with lessoptical distortion. It is thus possible to obtain a natural sense ofperspective when a vehicle or the like behind is visually recognized andto obtain satisfactory visibility with reduced degradation of imagequality.

Note that since similar effects can be obtained as long as theprojection property y(θ) satisfies the condition of Expression 1described above, the first embodiment is not limited to the projectionproperty illustrated in FIG. 2 . Note that in the first embodiment, theoptical system 10 having the projection property y(θ) satisfying thecondition of Expression 1 described above may be referred to as adifferent-angle-of-view lens.

Returning to FIG. 1 , the image processing unit 30 develops the imagedsignal generated by the imaging unit 20, performs processing such aswide dynamic range (WDR) correction, gamma correction, look up table(LUT) processing, and distortion correction, and thus generates imagedata.

For example, the image processing unit 30 executes an image processingstep of generating the image data from the imaged signal obtained in theimaging step. Through the processing, it is possible to obtain an imagethat can be easily visually recognized when it is displayed on the rearmonitor 60 and the electronic rear-view mirror 70.

Note that while the optical system 10 has a small amount of distortionand can perform image recognition without distortion correction in thehigh-resolution region, it is possible to improve a recognition rate ofthe vehicle-behind-on-lateral-side recognition unit 80 throughdistortion correction in the low-resolution region. The image processedby the image processing unit 30 is supplied to the rear monitor 60, theimage cutting processing unit 40, and the vehicle-behind-on-lateral-siderecognition unit 80.

The image cutting processing unit 40 performs cutting of a part of theimage processed by the image processing unit 30. The cutting of theimage is performed by storing data of the image processed by the imageprocessing unit 30 in a memory such as a RAM and reading only data ofthe image to be cut.

In the first embodiment, the position to be cut includes a second regionR2 corresponding to an angle of view for imaging the side behind thevehicle itself at a wide angle at the time of backward traveling, theimage of which has been formed by the optical system 10. Also, theposition includes a first region R1 corresponding to the angle of viewof high resolution for observing the vehicle 300 on the side behind thevehicle itself at the time of ordinary traveling.

In the first embodiment, the first region R1 is a region to be cut fromthe inside of the high-resolution region 10 a. Therefore, since theimage of the first region R1 can be displayed without distortioncorrection, it is possible to be displayed and recognized with a smallamount of delay at the time of high-speed traveling or the like, and itis thus advantageously used for avoiding a risk at the time of emergencyand the like.

FIG. 3A is an image diagram illustrating the first region R1 and thesecond region R2.

The region to be displayed on the electronic rear-view mirror 70 is thefirst region R1, and the region to be displayed on the rear monitor 60is the second region R2.

The image cutting position changing unit 50 changes the cutting positionof the image from the first region R1 in FIG. 3A to a third region R3where the vehicle 310 behind on the lateral side can be displayed in acase where the vehicle-behind-on-lateral-side recognition unit 80detects the vehicle 310 behind on the right side with respect to thetraveling direction as in FIG. 3B. Then, image data of the third regionR3 is displayed on the electronic rear-view mirror 70.

The change in cutting position of the image is performed by changingdata to be read in the image data after the image processing performedby the image processing unit 30 which is stored in the memory such asthe RAM. Here, the image cutting position changing unit 50 functions asa display control unit for controlling display on the rear monitor 60and the electronic rear-view mirror 70.

In this manner, the image cutting position changing unit 50 causes theimage data of the predetermined cut region (third region R3) includingthe high-distortion region including another moving apparatus behind onthe lateral side to be displayed in a case where the another movingapparatus behind on the lateral side is detected in the firstembodiment. FIG. 3B is an image diagram illustrating the third region R3and the second region R2.

Although the third region R3 is assumed to be located on the right sidebehind the vehicle itself in FIG. 3B, the present invention is notlimited to the right side behind. For example, the image cuttingposition changing unit 50 regards a region on the left side behind thevehicle itself as the third region R3 in a case where thevehicle-behind-on-lateral-side recognition unit 80 detects a vehiclebehind on the left side. For example, the third region R3 is a regionincluding the high-distortion region including the another movingapparatus behind on the lateral side.

The image cutting position changing unit 50 may cause the image of thefirst region R1 and the image of the third region R3 to be displayed onthe electronic rear-view mirror 70 at the same time by a method such asPbyP or PinP. The image cutting position changing unit 50 may notify theuser of the fact that switching to the image of the third region R3 hasbeen performed using sound, switching animation, or the like when theimage of the first region R1 is switched to the image of the thirdregion R3.

The rear monitor 60 is a display unit such as a liquid crystal display,is disposed in a center console, for example, of the vehicle itself, andis used to display a navigation screen or the like at the time ofordinary traveling. Also, the rear monitor 60 can also display the imageof the second region R2 after processing such as distortion correctionperformed by the image processing unit 30 in response to a selectionoperation performed by the user.

The electronic rear-view mirror 70 is a display unit such as a liquidcrystal display and is disposed above a wind shield, for example, of thevehicle itself. Also, the electronic rear-view mirror 70 displays eitherthe image of the first region R1 and the image of the third region R3cut by the image cutting processing unit 40 or both the images at thesame time.

The vehicle-behind-on-lateral-side recognition unit 80 detects that thevehicle 310 on the lateral side behind the vehicle itself is present inthe second region R2 of the image processed by the image processing unit30 through image recognition using a learned model for recognizing avehicle behind on the lateral side obtained by learning learning dataincluding the image of the vehicle behind on the lateral side.

Note that detection is not performed with regard to the first region R1at that time. Here, the vehicle-behind-on-lateral-side recognition unit80 functions as a detection unit that detects another moving apparatuson the lateral side behind the moving apparatus through imagerecognition based on image data including a high-distortion region.

Then, in a case where the vehicle 310 on the lateral side behind thevehicle itself is detected, the vehicle-behind-on-lateral-siderecognition unit 80 notifies the image cutting position changing unit 50of the fact. In a case where vehicles 310 are detected on both the rightside and the left side behind the vehicle itself, the image cuttingposition changing unit 50 is notified of the fact that there is avehicle on the lateral side behind the vehicle itself corresponding tothe later detection, for example.

Also, if the vehicle-behind-on-lateral-side recognition unit 80frequently notifies the image cutting position changing unit 50 ofwhether or not there is a vehicle 310 on the lateral side behind thevehicle itself, and the cutting position is changed every time, thecutting position of the image is frequently changed, which may becomplicated. Therefore, the image cutting position changing unit 50 maynot change the cutting position until a specific period of time elapsesafter detection of presence of the vehicle 310 on the lateral sidebehind the vehicle itself and notification to the image cutting positionchanging unit 50.

Next, control for changing a cutting position of an image when thevehicle 310 on the lateral side behind the vehicle itself is detectedfrom an image acquired by the vehicle-behind-on-lateral-side recognitionunit 80 will be described in order.

The image processing unit 30 performs image processing on an imageacquired by the imaging unit 20 and transmits the image to thevehicle-behind-on-lateral-side recognition unit 80.

The vehicle-behind-on-lateral-side recognition unit 80 monitors whetheror not there is a vehicle in a region other than the first region R1 inthe second region R2 in the input image. In a case where thevehicle-behind-on-lateral-side recognition unit 80 detects that there isa vehicle in a region other than the first region R1 in the secondregion R2, the vehicle-behind-on-lateral-side recognition unit 80notifies the image cutting position changing unit 50 of the fact that ithas detected the vehicle.

The image cutting position changing unit 50 changes the cutting positionof the image cut by the image cutting processing unit 40 from the firstregion R1 to the position where the vehicle 310 behind on the lateralside can be displayed (the third region R3 in a case of a vehicle behindon a right side).

FIG. 4 illustrates a flowchart of control for the image cutting positionchanging unit 50 to change the cutting position of the image when thevehicle 310 on the lateral side behind the vehicle itself is detectedfrom the acquired image. Note that the operation in each stepillustrated in FIG. 4 is controlled by a computer included in the imageprocessing apparatus 200 executing a computer program stored in astorage medium.

In Step S1, the vehicle-behind-on-lateral-side recognition unit 80determines whether or not there is a vehicle 310 within a predetermineddistance on the lateral side behind the vehicle itself. Here, Step S1functions as a detection step for detecting another moving apparatusbehind on the lateral side.

In a case where the vehicle-behind-on-lateral-side recognition unit 80determines that there is a vehicle 310 within the predetermined distanceon the lateral side behind the vehicle itself in Step S1, the processingproceeds to Step S2.

Then, in Step S2, the image cutting position changing unit 50 changesthe image cutting position from the first region R1 to a position wherethe vehicle 310 as another moving apparatus behind on the lateral sidecan be displayed (the third region R3 in a case of a vehicle behind onthe right side) and displays the image on the electronic rear-viewmirror 70. Note that the third region R3 is a region including thehigh-distortion region including the another moving apparatus behind onthe lateral side.

In a case where the vehicle-behind-on-lateral-side recognition unit 80determines that there is no vehicle 310 within the predetermineddistance on the lateral side behind the vehicle itself in Step S1, theprocessing proceeds to Step S3. Then, in Step S3, the image cuttingposition changing unit 50 determines the first region R1 as the cuttingposition and causes the electronic rear-view mirror 70 to display theimage of the first region R1. Here, Steps S1 to S3 function as a displaycontrol step.

Note that it is assumed that the image cutting position changing unit 50has changed the image cutting position to the position where the vehicle310 behind on the lateral side can be displayed (the third region R3 inthe case of the vehicle behind on the right side) in Step S2 and thevehicle 310 has then moved from the lateral side behind the vehicleitself. In such a case, the vehicle-behind-on-lateral-side recognitionunit 80 determines that there is no vehicle 310 within the predetermineddistance on the lateral side behind the vehicle itself in Step S1.

Then, since there is no need to continue the changing to the positionwhere the vehicle 310 behind on the lateral side can be displayed, theimage cutting position changing unit 50 returns the image cuttingposition to the first region R1 in Step S3. Note that the changing speedfor returning the image cutting position from the third region to thefirst region R1 may be lower than the changing speed when the imagecutting position is changed from the first region R1 to the thirdregion.

In the first embodiment, the control for changing the image cuttingposition when the vehicle 310 within the predetermined distance on thelateral side behind the vehicle itself is detected from the acquiredimage has been described. It is thus possible to check the vehicle atthe part corresponding to the blind angle where it is difficult torecognize it with a side mirror, such as a lateral side behind thevehicle itself, without using an imaging device or a detection unitother than the imaging device for imaging the side behind the vehicleitself.

Second Embodiment

In the first embodiment, the vehicle 310 on the lateral side behind thevehicle itself is detected through image recognition of an imagecaptured by the imaging device mounted on the rear side of the vehicle.In a second embodiment, a method for detecting the vehicle 310 on thelateral side behind the vehicle itself with another sensor instead ofthe imaging device will be described.

FIG. 5 is a block diagram for explaining a configuration example of animage processing apparatus 201 according to the second embodiment. Theimage processing apparatus 201 according to the second embodimentincludes an optical system 10, an imaging unit 20, an image processingunit 31, an image cutting processing unit 40, an image cutting positionchanging unit 51, a rear monitor 60, an electronic rear-view mirror 70,and a vehicle-behind-on-lateral-side detection unit 90.

The vehicle-behind-on-lateral-side detection unit 90 is a detection unitsuch as an infrared sensor or a radar disposed on the rear side of thevehicle itself and detects a vehicle 310 within a predetermined distanceon a right side or a left side behind the vehicle itself. Thevehicle-behind-on-lateral-side detection unit 90 notifies the imagecutting position changing unit 51 of the fact that it has detected thatthere is a vehicle within the predetermined distance on the right sideor the left side behind the vehicle itself.

In a case where the vehicle-behind-on-lateral-side detection unit 90detects the vehicle 310 behind on the right side or the left side withinthe predetermined distance, the image cutting position changing unit 51changes the image cutting position from the first region R1 to apredetermined cutting position where the vehicle behind on the rightside or the left side can be displayed. Then, the image is displayed onthe electronic rear-view mirror 70.

Note that since configurations other than thevehicle-behind-on-lateral-side detection unit 90 and the image cuttingposition changing unit 51 are the same as those in the first embodiment,description thereof will be omitted.

As described above, according to the second embodiment, it is possibleto detect the vehicle 310 within the predetermined distance on thelateral side behind the vehicle itself with a detection unit such as aninfrared sensor or a radar disposed on the side behind the vehicleitself and to check the vehicle at the part corresponding to the blindangle on the lateral side behind the vehicle itself in the image.

Third Embodiment

In the first and second embodiments, the control for changing the imagecutting position when the vehicle 310 is detected within thepredetermined distance on the lateral side behind the vehicle itself hasbeen described. There may be a case where it is easier to check avehicle at a part corresponding to a blind angle where it is difficultto recognize the vehicle with a side mirror, such as a lateral sidebehind the vehicle itself by changing the size of the cut imagedepending on the size of the vehicle 310 within the predetermineddistance on the lateral side behind the vehicle itself.

In a case where the vehicle 310 within the predetermined distance on thelateral side behind the vehicle itself is a large truck, for example,the entire vehicle cannot be recognized depending on the size of the cutimage, and an increase in size of the image to enable the entire vehicleto be recognized can thus further contribute to safety check. Therefore,according to the third embodiment, the size of the cut image is changedin accordance with the size of the detected vehicle 310 within thepredetermined distance on the lateral side behind the vehicle itself.

FIG. 6 is a block diagram for explaining a configuration example of animage processing apparatus 202 according to a third embodiment. Theimage processing apparatus 202 includes an optical system 10, an imagingunit 20, an image processing unit 30, an image cutting processing unit41, an image cutting position changing unit 50, a rear monitor 60, anelectronic rear-view mirror 70, and a vehicle-behind-on-lateral-siderecognition unit 81.

The vehicle-behind-on-lateral-side recognition unit 81 detects thatthere is a vehicle 310 within a predetermined distance on the lateralside behind the vehicle itself through image recognition on an imageprocessed by the image processing unit 30 using a learned model forrecognizing a vehicle behind on the lateral side obtained by learninglearning data including an image of the vehicle behind on the lateralside. Then, the vehicle-behind-on-lateral-side recognition unit 81notifies the image cutting position changing unit 50 of the imagerecognition result.

Moreover, it is possible to detect the size of the vehicle 310 withinthe predetermined distance on the lateral side behind the vehicle itselfusing the learned model for recognizing the size of the vehicle behindon the lateral side within the predetermined distance, which has beenobtained by learning the learning data including the image of thevehicle behind on the lateral side that may have various sizes withinthe predetermined distance. Then, the vehicle-behind-on-lateral-siderecognition unit 81 notifies the image cutting processing unit 41 ofsize information of the vehicle 310 within the predetermined distance onthe lateral side behind the vehicle itself.

As the size of the vehicle 310 within the predetermined distance on thelateral side behind the vehicle itself detected by thevehicle-behind-on-lateral-side recognition unit 81, information as anarbitrary size with which the vehicle 310 on the lateral side behind thevehicle itself can be imaged may be provided as a notification, orinformation may be provided as a notification by categorizing the sizeinto a plurality of levels such as large, middle, and small. Note thatin a case where vehicles 310 are detected within the predetermineddistance on both the right side and the left side behind the vehicleitself, a notification that there is a vehicle within the predetermineddistance on the lateral side behind the vehicle itself corresponding tothe later detection is provided to the image cutting position changingunit 50.

The image cutting processing unit 41 performs cutting of a part of theimage processed by the image processing unit 30. The cutting of theimage is performed by storing data of the image after image processingperformed by the image processing unit 30 in a memory such as a RAM andreading only data of the image to be cut.

The position to be cut is a first region R1 corresponding to a narrowangle of view of high resolution for observing the vehicle 300 on theside behind the vehicle itself at the time of ordinary traveling in asecond region corresponding to an angle of view for imaging the sidebehind the vehicle itself at a wide angle at the time of backwardtraveling, the image of which has been formed by the optical system 10.Also, the size of a third region R3 to be cut is changed in accordancewith the information on the size of the vehicle 310 within thepredetermined distance on the lateral side behind the vehicle itselfprovided as a notification from the vehicle-behind-on-lateral-siderecognition unit 81.

Note that since configurations other than thevehicle-behind-on-lateral-side recognition unit 81 and the image cuttingprocessing unit 41 are the same as those in the first embodiment,description thereof will be omitted.

Since the size of the image to be cut is changed in accordance with thesize of the vehicle 310 within the predetermined distance on the lateralside behind the vehicle itself in the third embodiment, it is possibleto check the entire vehicle regardless of the size of the vehicle 310within the predetermined distance on the lateral side behind the vehicleitself.

Since the image data capacity increases by increasing the size of theimage to be cut by the image cutting processing unit 41, the image datacapacity may be reduced by lowering a frame rate of the image ordownscaling the image.

Note that in a case where it is not desired to lower the frame rate ofthe image or downscale the image, control for changing the size of theimage to be cut may be able to be selectively turned off in accordancewith the size of the vehicle 310 on the lateral side behind the vehicleitself.

Fourth Embodiment

In a fourth embodiment, a method for checking a part corresponding to ablind angle such as a lateral side behind the vehicle itself, which isdifficult to be checked with a side mirror, in an image when the vehicleitself is about to turn right or left or change lanes will be described.

FIG. 7 is a block diagram for explaining a configuration example of animage processing apparatus 203 according to a fourth embodiment. Theimage processing apparatus 203 includes an optical system 10, an imagingunit 20, an image processing unit 30, an image cutting processing unit40, an image cutting position changing unit 52, a rear monitor 60, anelectronic rear-view mirror 70, and a direction indicating operationdetection unit 100.

The direction indicating operation detection unit 100 detects anoperation of turning on a direction indicator such as a turn (signal)switch or a turn (signal) lever, that is performed when the vehicleturns right or left or changes lanes with a temporary change intraveling direction and a direction of the direction indication andnotifies the image cutting position changing unit 52 of the operationand the direction. Here, the direction indicating operation detectionunit 100 functions as a direction detection unit that executes adirection detection step for detecting a change in traveling directionof the moving apparatus by detecting the operation direction of thedirection indicator provided in the moving apparatus.

The image cutting position changing unit 52 changes the image cuttingposition cut by the image cutting processing unit 40 to a predeterminedcutting position (fourth region R4) where the lateral side behind in thedirection in which the direction indicator is turned on can be imagedand displays the predetermined cutting position on the electronicrear-view mirror 70.

For example, the image cutting position changing unit 50 causes imagedata of the predetermined cut region (fourth region R4) including ahigh-distortion region on the side of the changed traveling direction tobe displayed in a case where a change in traveling direction of themoving apparatus is detected in the fourth embodiment.

In a case where the right or left direction indicator is turned on, forexample, the cutting position is changed to the predetermined cuttingposition (fourth region R4) such that the part corresponding to a blindangle which is difficult to be checked with a side mirror on the rightor left side behind the vehicle itself can be imaged.

At this time, the predetermined cutting position (fourth region R4)includes the high-distortion region on the side of the travelingdirection. The change in the image cutting position is performed bychanging data to be read in data of the image processed by the imageprocessing unit 30 and stored in a memory such as a RAM.

Note that since configurations other than the image cutting positionchanging unit 52 and the direction indicating operation detection unit100 are the same as those in the first embodiment, description thereofwill be omitted.

FIG. 8 illustrates a flowchart of control for changing the image cuttingposition when an operation of the direction indicator is detected. Notethat each step illustrated in FIG. 8 is controlled by a computerincluded in the image processing apparatus 203 executing a computerprogram stored in a storage medium.

In Step S7, the direction indicating operation detection unit 100determines whether or not the operation of turning on the directionindicator has been performed and a direction in which the directionindicator has been turned on. In a case where it is determined that theoperation of turning on the direction indicator has been performed, thedirection indicating operation detection unit 100 notifies the imagecutting position changing unit 52 of the fact that the operation ofturning on the direction indicator has been performed and of thedirection in which the direction indicator has been turned on.

In a case where the direction indicating operation detection unit 100determines that the operation of turning on the direction indicator hasbeen performed in Step S7, the processing proceeds to Step S8. Then, inStep S8, the image cutting position changing unit 52 changes the cutimage from the first region R1 to the predetermined cutting position(fourth region R4) where the lateral side behind the vehicle itself inthe direction in which the direction indicator has been turned on can beimaged.

Then, the image cutting position changing unit 52 displays the image atthe predetermined cutting position (fourth region R4) after the changeon the electronic rear-view mirror 70.

In a case where the direction indicating operation detection unit 100determines that the operation of turning on the direction indicator hasnot been performed in Step S7, the processing proceeds to Step S9. Then,in Step S9, the image cutting position changing unit 52 displays thecutting position of the image cut by the image cutting processing unit40 as the first region R1 on the electronic rear-view mirror 70.

In a case where the vehicle itself completes turning right or left orchanging of lanes after the direction indicating operation detectionunit 100 detects the operation of the direction indicator and changesthe image cutting position, there is no need to continue the changing ofthe image cutting position.

Therefore, the image cutting position changing unit 52 returns the imagecutting position to the first region R1. The fact that the vehicleitself has completed turning right or left or changing of lanes isdetermined by not detecting the operation of the direction indicator.

In this manner, since the image cutting position is changed when theoperation of the direction indicator is detected in the fourthembodiment, it is possible to reliably check, in the image, the partcorresponding to the blind angle which is difficult to be checked withthe side mirror, such as the right and left sides behind the vehicleitself when the vehicle itself is about to turn right or left or changelanes.

Fifth Embodiment

Examples in which a part corresponding to a blind angle that isdifficult to be checked with a side mirror, such as a lateral sidebehind the vehicle itself, is checked with the imaging device on theside behind the vehicle itself have been described in the first tofourth embodiments. The imaging device on the side behind the vehicleitself is originally for checking the condition behind (right behind)the vehicle itself.

Therefore, in a case where there is a vehicle 300 on the side behind thevehicle itself, there may be a case where it is desired to continuouslycheck a state behind the vehicle itself without changing the imagecutting position even when there is the vehicle 310 on the lateral sidebehind the vehicle itself. In the fifth embodiment, an example in whichwhether or not to change the image cutting position is controlled in acase where there is a vehicle 300 on the side behind the vehicle itselfin accordance with user setting will be described.

FIG. 9 is a block diagram for explaining a configuration example of animage processing apparatus 204 according to the fifth embodiment. Theimage processing apparatus 204 includes an optical system 10, an imagingunit 20, an image processing unit 32, an image cutting processing unit40, and an image cutting position changing unit 53. Furthermore, theimage processing apparatus 204 includes a rear monitor 60, an electronicrear-view mirror 70, a vehicle-behind-on-lateral-side recognition unit80, a vehicle-behind recognition unit 110, and an image cutting settingsaving unit 120.

The image processing unit 32 performs processing such as wide dynamicrange (WDR) correction, gamma correction, LUT processing, and distortioncorrection on digital data transmitted from the imaging unit 20. Throughthe processing, it becomes easier to visually recognize the digital datadisplayed on the rear monitor 60 and the electronic rear-view mirror 70,and a recognition rate of the vehicle-behind-on-lateral-side recognitionunit 80 is improved.

The image processed by the image processing unit 32 is input to the rearmonitor 60, the image cutting processing unit 40, thevehicle-behind-on-lateral-side recognition unit 80, and thevehicle-behind recognition unit 110.

The vehicle-behind recognition unit 110 detects, in the image after theimage processing performed by the image processing unit 32, that thereis a vehicle 300 on the side behind the vehicle itself using a learnedmodel for recognizing a vehicle behind obtained by learning learningdata including the image of the vehicle behind and notifies the imagecutting position changing unit 53 of the fact.

The image cutting setting saving unit 120 saves setting regardingwhether or not to cause the image cutting position changing unit 53 tochange the image cutting position and notifies the image cuttingposition changing unit 53 of the setting content in a case where thevehicle-behind recognition unit 110 detects that there is a vehicle 300on the side behind the vehicle itself. Here, the image cutting settingsaving unit 120 functions as a setting unit for setting whether or notto cause image data in the predetermined cut region to be displayed.

Setting of causing the image cutting position changing unit 53 to changethe image cutting position in a case where the vehicle-behindrecognition unit 110 detects that there is a vehicle 300 on the sidebehind the vehicle itself will be defined as change setting below.Setting of not causing the image cutting position changing unit 53 tochange the image cutting position in a case where the vehicle-behindrecognition unit 110 detects the vehicle 300 on the side behind thevehicle itself will be defined as non-change setting below.

The image cutting position changing unit 53 receives a notification thatthe vehicle-behind-on-lateral-side recognition unit 80 has detected thevehicle 310 behind on the lateral side, a notification that thevehicle-behind recognition unit 110 has detected the vehicle 300 on theside behind the vehicle itself, and a notification of setting saved inthe image cutting setting saving unit 120. Then, the image cuttingposition changing unit 53 determines whether or not to change the imagecutting position from the first region R1 to the position where thevehicle 310 behind on the lateral side can be displayed on the basis ofthe information in the notifications.

The image cutting position changing unit 53 displays either the image ofthe first region R1 or the image of the third region R3 after thechanging of the cutting position or both the images on the electronicrear-view mirror 70 on the basis of the determination result. Details ofthe determination will be described in the flowchart in FIG. 10 . Thechange in image cutting position is performed by changing data to beread in the image data after image processing performed by the imageprocessing unit 32 stored in the memory such as an RAM.

Note that since configurations other than the image processing unit 32,the image cutting position changing unit 53, the vehicle-behindrecognition unit 110, and the image cutting setting saving unit 120 arethe same as those in the first embodiment, description thereof will beomitted.

Next, control for determining whether or not to change the image cuttingposition in accordance with setting of the image cutting setting savingunit 120 will be described in order. The user saves the aforementionedchange setting or non-change setting in the image cutting setting savingunit 120.

In a case where the change setting is saved in the image cutting settingsaving unit 120, the image cutting position changing unit 53 changes theimage cutting position once the vehicle-behind-on-lateral-siderecognition unit 80 detects the vehicle 310 within the predetermineddistance on the lateral side behind the vehicle itself regardless ofwhether the vehicle 300 on the side behind the vehicle itself has beendetected.

In a case where the non-change setting is saved in the image cuttingsetting saving unit 120, and in a case where the vehicle 300 on the sidebehind the vehicle itself has been detected, the image cutting positionchanging unit 53 does not change the image cutting position even if thevehicle-behind-on-lateral-side recognition unit 80 detects the vehicle310 within the predetermined distance on the lateral side behind thevehicle itself.

FIG. 10 illustrates a flowchart for controlling whether or not to changethe image cutting position in accordance with setting of the imagecutting setting saving unit 120. Note that each step illustrated in FIG.10 is controlled by a computer included in the image processingapparatus 204 executing a computer program stored in a storage medium.

In Step S10, the vehicle-behind recognition unit 110 determines thatthere is a vehicle 300 on the side behind the vehicle itself.

In a case where the vehicle-behind recognition unit 110 determines thatthere is a vehicle 300 on the side behind the vehicle itself in StepS10. the vehicle-behind-on-lateral-side recognition unit 80 determinesthat there is a vehicle 310 within the predetermined distance on thelateral side behind the vehicle itself in Step S11.

In a case where the vehicle-behind-on-lateral-side recognition unit 80determines that there is no vehicle 310 within the predetermineddistance on the lateral side behind the vehicle itself in Step S11, theimage cutting position changing unit 50 does not change the imagecutting position.

In a case where the vehicle-behind recognition unit 110 determines thatthere is no vehicle 300 on the side behind the vehicle itself in StepS10, the processing in Step S13 is performed.

In a case where the vehicle-behind-on-lateral-side recognition unit 80determines that there is a vehicle 310 within the predetermined distanceon the lateral side behind the vehicle itself in Step S11, whether ornot the non-change setting is saved in the image cutting setting savingunit 120 is determined in Step S12.

In a case where the change setting is saved, the image cutting positionchanging unit 50 changes the cutting position in Step S15. In a casewhere the non-change setting is saved in the image cutting settingsaving unit 120 in Step S12, the image cutting position changing unit 50does not change the cutting position in Step S13.

Since whether or not to change the image cutting position is determinedin accordance with the setting of the image cutting setting saving unit120 in the fifth embodiment, it is possible to prevent the image cuttingposition from being changed despite the intension of the user.

Note that the unit for detecting the vehicle on the lateral side behindthe vehicle itself is the vehicle-behind-on-lateral-side recognitionunit 80 in the fifth embodiment, another sensor may perform thedetection as in the second embodiment.

Although the example in which the cutting position change setting andnon-change setting can be selected in a case where the vehicle 310 isdetected on the lateral side behind the vehicle itself has beendescribed in the fifth embodiment, the cutting position change settingand non-change setting may be able to be selected in a case where anoperation of turning on the direction indicator is performed. Forexample, the image cutting setting saving unit 120 can selectively setwhether or not to cause image data of a predetermined cut region (thirdregion R3 or the fourth region R4) to be displayed.

Seventh Embodiment

The moving apparatus (vehicle itself or another vehicle) according tothe aforementioned embodiments is not limited to an automobile and maybe any movable apparatus such as a motorcycle, a bicycle, a wheelchair,a ship, an airplane, a robot, or a drone.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

In addition, as a part or the whole of the control according to theembodiments, a computer program realizing the function of theembodiments described above may be supplied to the image processingapparatus through a network or various storage media. Then, a computer(or a CPU, an MPU, or the like) of the image processing apparatus may beconfigured to read and execute the program. In such a case, the programand the storage medium storing the program configure the presentinvention.

This application claims the benefit of Japanese Patent Application No.2021-177623 filed on Oct. 29, 2021, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing apparatus comprising: animaging device including an optical system for forming an optical imageincluding a low-distortion region and a high-distortion region on alight receiving surface and configured to generate an image signal of aside behind a moving apparatus; and at least one processor or circuitconfigured to function as: an image processing unit configured togenerate image data from the image signal generated by the imagingdevice, a detection unit configured to detect another moving apparatuson a lateral side behind the moving apparatus, and a display controlunit configured to cause the image data in a predetermined cut regionincluding the high-distortion region including the moving apparatusbehind on the lateral side to be displayed in a case where the anothermoving apparatus behind on the lateral side is detected by the detectionunit.
 2. The image processing apparatus according to claim 1, wherein ina case where the detection unit does not detect the another movingapparatus behind on the lateral side, the image data in a first regioncorresponding to the low-distortion region in the image data is causedto be displayed.
 3. The image processing apparatus according to claim 1,wherein in a case where a focal distance of the optical systems isdefined as f, a half angle of view is defined as θ, an image height onan image plane is defined as y, and a projection property representing arelationship between the image height y and the half angle of view θ isdefined as y(θ), y(θ) in the low-distortion region is greater than f×θand is different from the projection property in the high-distortionregion.
 4. The image processing apparatus according to claim 3, whereinthe low-distortion region is configured to have a projection propertythat is approximated to a center projection method (y=f×tan θ) or anequidistant projection method (y=f×θ).
 5. The image processing apparatusaccording to claim 3, wherein in a case where θ max is defined as amaximum half angle of view that the optical system has, the imageprocessing apparatus is configured to satisfy1<f×sin(θ max)/y(θ max)≤1.9.
 6. The image processing apparatus accordingto claim 1, wherein the detection unit is able to detect a size of theanother moving apparatus behind on the lateral side, and the displaycontrol unit changes a size of the predetermined cut region inaccordance with the size of the another moving apparatus behind on thelateral side detected by the detection unit.
 7. The image processingapparatus according to claim 1, wherein the detection unit detects theanother moving apparatus behind on the lateral side through imagerecognition based on the image data including the high-distortionregion.
 8. The image processing apparatus according to claim 1, whereinthe detection unit includes an infrared sensor or a radar.
 9. The imageprocessing apparatus according to claim 1, wherein the at least oneprocessor or circuit is further configured to function as: a settingunit configured to set whether or not to cause the display control unitto display the image data in the predetermined cut region.
 10. The imageprocessing apparatus according to claim 1, wherein the at least oneprocessor or circuit is further configured to function as: a directiondetection unit configured to detect a change in traveling direction ofthe moving apparatus, the display control unit causes the image data ina predetermined cut region including the high-distortion region on aside of the changed traveling direction to be displayed in a case wherethe direction detection unit detects a change in traveling direction ofthe moving apparatus.
 11. An image processing apparatus comprising: animaging device including an optical system for forming an optical imageincluding a low-distortion region and a high-distortion region on alight receiving surface and configured to generate an image signal of aside behind a moving apparatus; and at least one processor or circuitconfigured to function as: an image processing unit configured togenerate image data from the image signal generated by the imagingdevice, a direction detection unit configured to detect a change intraveling direction of the moving apparatus, and a display control unitconfigured to cause the image data in a predetermined cut regionincluding the high-distortion region on a side of the changed travelingdirection to be displayed in a case where a change in travelingdirection of the moving apparatus is detected by the direction detectionunit.
 12. The image processing apparatus according to claim 11, whereinin a case where the direction detection unit does not detect a change intraveling direction of the moving apparatus, the image data in a firstregion corresponding to the low-distortion region in the image data iscaused to be displayed.
 13. The image processing apparatus according toclaim 11, wherein in a case where a focal distance of the opticalsystems is defined as f, a half angle of view is defined as θ, an imageheight on an image plane is defined as y, and a projection propertyrepresenting a relationship between the image height y and the halfangle of view θ is defined as y(θ), y(θ) in the low-distortion region isgreater than f×θ and is different from the projection property in thehigh-distortion region.
 14. The image processing apparatus according toclaim 13, wherein the low-distortion region is configured to have aprojection property that is approximated to a center projection method(y=f×tan θ) or an equidistant projection method (y=f×θ).
 15. The imageprocessing apparatus according to claim 13, wherein in a case where θmax is defined as a maximum half angle of view that the optical systemhas, the image processing apparatus is configured to satisfy1<f×sin(θ max)/y(θ max)≤1.9.
 16. The image processing apparatusaccording to claim 11, wherein the direction detection unit detects anoperation direction of a direction indicator provided in the movingapparatus.
 17. The image processing apparatus according to claim 11,wherein the at least one processor or circuit is further configured tofunction as: a setting unit configured to set whether or not to causethe display control unit to display the image data in the predeterminedcut region.
 18. An image processing method comprising: generating animage signal of a side behind a moving apparatus using an imaging deviceincluding an optical system configured to form an optical imageincluding a low-distortion region and a high-distortion region on alight receiving surface; generating image data from the image signal ofthe side behind the moving apparatus; detecting another moving apparatuson a lateral side behind the moving apparatus; and causing the imagedata in a predetermined cut region including the high-distortion regionincluding the another moving apparatus behind on the lateral side to bedisplayed in a case where the another moving apparatus behind on thelateral side is detected in the detecting of the moving apparatus. 19.An image processing method comprising: generating an image signal of aside behind a moving apparatus using an imaging device including anoptical system configured to form an optical image including alow-distortion region and a high-distortion region on a light receivingsurface; generating image data from the image signal of the side behindthe moving apparatus; detecting a change in traveling direction of themoving apparatus; and causing the image data in a predetermined cutregion including the high-distortion region on a side of the changedtraveling direction to be displayed in a case where a change intraveling direction of the moving apparatus is detected in the detectingof the change in traveling direction.
 20. A non-transitorycomputer-readable storage medium configured to store a computer programcomprising instructions for executing following processes: generating animage signal of a side behind a moving apparatus using an imaging deviceincluding an optical system configured to form an optical imageincluding a low-distortion region and a high-distortion region on alight receiving surface; generating image data from the image signal ofthe side behind the moving apparatus; detecting another moving apparatuson a lateral side behind the moving apparatus; and causing the imagedata in a predetermined cut region including the high-distortion regionincluding the another moving apparatus behind on the lateral side to bedisplayed in a case where the another moving apparatus behind on thelateral side is detected in the detecting of the moving apparatus.
 21. Anon-transitory computer-readable storage medium configured to store acomputer program comprising instructions for executing followingprocesses: generating an image signal of a side behind a movingapparatus using an imaging device including an optical system configuredto form an optical image including a low-distortion region and ahigh-distortion region on a light receiving surface; generating imagedata from the image signal of the side behind the moving apparatus;detecting a change in traveling direction of the moving apparatus; andcausing the image data in a predetermined cut region including thehigh-distortion region on a side of the changed traveling direction tobe displayed in a case where a change in traveling direction of themoving apparatus is detected in the detecting of the change in travelingdirection.